Estimation of Software Size and Effort Distributions Using Paired Ratio Comparison Matrices

Recently the Software Quality Improvement (SQI) Project has been formed at NASA’s Jet Propulsion Laboratory (JPL). SQI will enable and promote software best practices, and leverage JPL experience in software engineering in support of major software projects, throughout the entire software life-cycle. The goal of the SQI Project is to establish an operational software improvement program that results in the continuous measurable improvement at JPL. Its objectives include improving cost and schedule predictability, improving the quality of missioncritical software, reducing software defect rates during testing and operations, increasing software development productivity, promoting software reuse, and reducing project start-up time. In this paper we will document one of several approaches being introduced at JPL to improve its ability to improve cost estimation accuracy early in the project life-cycle. We were recently confronted with a problem where a cost estimate was required for a piece of mission critical software. The technical staff did not trust cost models, and they had numerous sources of potential risk and uncertainty associated with the next build. The task wanted to estimate the costs of its next delivery based on the cost of its current delivery. However, the task had virtually no retrievable historical data fiom their previous builds, for either effort or software size. To help them formalize their expert judgment-based estimates, we considered using Galorath Corporation’s SEER-SSM, which is an adaptation of the paired ratio comparison matrixes described in Saaty’s Analytical Hierarchy Process. While this approach is very attractive, a number of problems arose in that we had multiple reference projects, the technical staff wanted to provide ranges for the ratio comparisons, and we wanted a distribution for the size andor effort estimates rather than a point value. We also had to use the method to help us reconstruct the actuals as well as estimate the next build. In this paper we describe how the pairwise comparison technique is a general purpose estimation approach for capturing expert judgment and can be relatively easily implemented using Microsoft ExcelO, if the geometric mean method is used to derive the ratio vector. We document how this approach can be further generalized to a probabilistic version using Monte Carlo methods to produce estimates of size and effort distributions. The probabilistic pairwise comparison technique enables the estimator to systematically incorporate both estimation uncertainty as well as any uncertainty that arises from using multiple historical analogies as reference modules. In addition to describing the methodology, we will also describe the results of the case study.